Looking away determination device, looking away determination system, looking away determination method, and storage medium

ABSTRACT

A looking away determination device includes: a determination unit that determines that a driver is in a looking away state when a proportion of an image in which a face of the driver is not detected with respect to a plurality of images obtained by imaging the driver is equal to or greater than a first predetermined value, the first predetermined value being determined based on driving state information that indicates a state during driving by the driver.

TECHNICAL FIELD

The present invention relates to a looking away determination device, alooking away determination system, a looking away determination method,and a storage medium.

BACKGROUND ART

Patent Document 1 discloses a technique for detecting looking away of adriver operating a moving body such as an automobile.

PRIOR ART DOCUMENTS Patent Documents

[Patent Document 1] PCT International Publication No. WO 2016/052507

SUMMARY OF THE INVENTION Problem to be Solved by the Invention

In the above-described technique for determining looking away duringdriving, looking away detection is performed on the basis of theline-of-sight of the face that appears in a captured image. However,when a face does not appear in the captured image, looking awaydetermination is not performed at all. It is therefore required toappropriately perform looking away determination according to thedriving state even if a face does not appear.

An example object of the present invention is to provide a looking awaydetermination device, a looking away determination system, a lookingaway determination method, and a storage medium that solves theaforementioned issue.

Means for Solving the Problem

According to a first example aspect of the present invention, a lookingaway determination device includes: a determination unit that determinesthat a driver is in a looking away state when a proportion of an imagein which a face of the driver is not detected with respect to aplurality of images obtained by imaging the driver is equal to orgreater than a first predetermined value, the first predetermined valuebeing determined based on driving state information that indicates astate during driving by the driver.

According to a second example aspect of the present invention, a lookingaway determination system includes an imaging device and a looking awaydetermination device. The looking away determination device includes adetermination unit that determines that a driver is in a looking awaystate when a proportion of an image in which a face of the driver is notdetected with respect to a plurality of images obtained by imaging thedriver by the imaging device is equal to or greater than a firstpredetermined value, the first predetermined value being determinedbased on driving state information that indicates a state during drivingby the driver.

According to a third example aspect of the present invention, a lookingaway determination method includes: determining that a driver is in alooking away state when a proportion of an image in which a face of thedriver is not detected with respect to a plurality of images obtained byimaging the driver is equal to or greater than a first predeterminedvalue, the first predetermined value being determined based on drivingstate information that indicates a state during driving by the driver.

According to a fourth example aspect of the present invention, a storagemedium stores a program that causes a computer to execute: determiningthat a driver is in a looking away state when a proportion of an imagein which a face of the driver is not detected with respect to aplurality of images obtained by imaging the driver is equal to orgreater than a first predetermined value, the first predetermined valuebeing determined based on driving state information that indicates astate during driving by the driver.

Effect of the Invention

According to an example embodiment of the present invention, when it isdetermined that a face does not appear based on a captured image, it ispossible to determine whether or not there is looking away in accordancewith the driving state.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a drawing showing a driving state monitoring system accordingto an example embodiment of the present invention.

FIG. 2 is a hardware configuration diagram of a looking awaydetermination device according to the example embodiment of the presentinvention.

FIG. 3 is a function block diagram of the looking away determinationdevice according to the example embodiment of the present invention.

FIG. 4 is diagram showing a hardware configuration of a drive recorderaccording to the example embodiment of the present invention.

FIG. 5 is a function block diagram of a control device of the driverecorder according to the example embodiment of the present invention.

FIG. 6 is a diagram showing the processing flow of the drive recorderaccording to the example embodiment of the present invention.

FIG. 7 is a diagram showing the processing flow of the looking awaydetermination device according to the example embodiment of the presentinvention.

FIG. 8A is a diagram showing an example of the condition range of thefacial direction or line-of-sight direction for determining looking awayin the looking away determination device according to the exampleembodiment of the present invention.

FIG. 8B is a diagram showing an example of the condition range of thefacial direction or line-of-sight direction for determining looking awayin the looking away determination device according to the exampleembodiment of the present invention.

FIG. 8C is a diagram showing an example of the condition range of thefacial direction or line-of-sight direction for determining looking awayin the looking away determination device according to the exampleembodiment of the present invention.

FIG. 9 is a drawing showing a configuration of a looking awaydetermination device according to another example embodiment of thepresent invention.

EXAMPLE EMBODIMENT

Hereinbelow, a looking away determination device according to exemplaryexample embodiments of the present invention will be described withreference to the drawings.

FIG. 1 is a diagram showing a driving state monitoring system accordingto an example embodiment of the present invention.

As shown in FIG. 1, a driving state monitoring system 100 includes alooking away determination device 1 and a drive recorder 2 that is oneform of a driving state sensing device. The looking away determinationdevice 1 and the drive recorder 2 are connected via a wirelesscommunication network or a wired communication network. The driverecorder 2 is provided in a vehicle as an example. The looking awaydetermination device 1 is communicatively connected to the driverecorders 2 respectively installed in a plurality of vehicles that arerunning throughout a city.

FIG. 2 is a hardware configuration diagram of the looking awaydetermination device 1.

As shown in FIG. 2, the looking away determination device 1 is acomputer including hardware such as a CPU (Central Processing Unit) 101,a ROM (Read Only Memory) 102, a RAM (Random Access Memory) 103, adatabase 104, and a communication module 105.

FIG. 3 is a function block diagram of the looking away determinationdevice 1.

The looking away determination device 1 is activated when the power isturned on, and executes a looking away determination program stored inadvance. As a result, the looking away determination device 1 exhibitsat least the functions of a control unit 11, an information acquisitionunit 12, a looking away start determination unit 13, a looking awaydetermination unit (determination unit) 14, and a determination resultoutput unit 15.

The control unit 11 controls other function units.

The information acquisition unit 12 acquires information transmittedfrom the drive recorder 2, such as captured images, vehicle information,weather information, and acceleration information.

The looking away start determination unit 13 determines whether to startlooking away determination.

The looking away determination unit 14 determines whether or not a facehas been detected on the basis of a captured image. The looking awaydetermination unit 14 determines that there is looking away state when aface cannot be detected a predetermined proportion or more per unit timespecified on the basis of driving state information acquired duringdriving. The looking away determination unit 14 determines whether ornot the facial direction is within a predetermined condition range onthe basis of the captured image, and determines that there is a lookingaway state when the time during which the facial direction is not withinthe predetermined condition range determined on the basis of the drivingstate information is equal to or greater than the predeterminedproportion per unit time. In addition, the looking away determinationunit 14 determines whether or not the line-of-sight direction is withina predetermined condition range on the basis of the captured image, anddetermines that there is a looking away state when the time during whichthe line-of-sight direction is not within the predetermined conditionrange determined on the basis of the driving state information is equalto or greater than a predetermined proportion per unit time. The lookingaway determination unit 14 determines the condition range to be acondition range extended in the right direction when the driving stateinformation indicates that the course is changed to the right direction.The looking away determination unit 14 determines the condition range tobe a condition range extended in the left direction when the drivingstate information indicates that the course is changed to the leftdirection.

That is, the looking away determination unit 14 determines a driver isin a looking away state when the proportion of images in which the faceof the driver is not detected with respect to a plurality of imagesobtained by imaging the driver (during a unit time (predeterminedperiod)) is equal to or greater than a first predetermined value thathas been determined on the basis of driving state information indicatingthe state at the time of driving by the driver. The looking awaydetermination unit 14 determines a driver is in a looking away statewhen the proportion of images in which the facial direction of thedriver is not within a facial direction condition range determined onthe basis of the driving state information is equal to or greater than asecond predetermined value. The looking away determination unit 14determines a driver is in a looking away state when the proportion ofimages in which the line-of-sight direction of the driver is not withina line-of-sight direction condition range determined on the basis of thedriving state information is equal to or greater than a thirdpredetermined value. Each unit time (predetermined period) may be thesame or different. Moreover, each predetermined value may be the same ordifferent.

The determination result output unit 15 outputs the looking awaydetermination result.

FIG. 4 is a diagram showing a hardware configuration of the driverecorder 2.

The drive recorder 2 includes an acceleration sensor 21, a communicationdevice 22, a camera 23, a control device 24, and a storage device 25.The acceleration sensor 21 detects the acceleration of the vehicle. Thecommunication device 22 is communicatively connected to the looking awaydetermination device 1. The camera 23 captures images of the outside orthe inside of the vehicle to generate moving images and still images.

The control device 24 controls each function of the drive recorder 2.The storage device 25 stores moving images, still images, accelerationdetected by the acceleration sensor 21, other information obtained fromoutside the drive recorder 2, and the like. The drive recorder 2 iscommunicatively connected to the looking away determination device 1 viaa base station or the like. The control device 24 of the drive recorder2 is a computer including a CPU, ROM, RAM and the like.

FIG. 5 is a function block diagram of the control device 24 provided inthe drive recorder 2.

The control device 24 executes the control program when the driverecorder starts up. Thereby, the control device 24 is capable offunctioning as a vehicle information acquisition unit 241, a weatherinformation acquisition unit 242, an acceleration informationacquisition unit 243, a captured image acquisition unit 244, a drivingstate data transmission unit 245, and a captured image transmission unit246.

FIG. 6 is a diagram showing the processing flow of the drive recorder 2.

Next, the processing flow of the driving state monitoring system will beexplained step by step.

First, the transmission process for the driving state information in thedrive recorder 2 will be described.

When the electrical system of the vehicle is activated, the driverecorder 2 starts operation (Step S101). The acceleration sensor 21 ofthe drive recorder 2 starts sensing acceleration of the vehicle afterthe drive recorder 2 is started (Step S102). Further, the camera 23starts imaging the inside of the vehicle and the outside of the vehicle(Step S103). The camera 23 includes a vehicle inside lens and a vehicleoutside lens. The camera 23 uses the vehicle inside lens to capture animage of an object in the direction toward the driver's face in thevehicle (a rearward scene in the traveling direction (including thedriver's face)). The camera 23 uses the vehicle outside lens to capturean image of an object in the traveling direction outside the vehicle.

Then, during operation of the drive recorder 2, the vehicle informationacquisition unit 241 of the control device 24 acquires vehicleinformation (Step S104). The vehicle information acquired by the vehicleinformation acquisition unit 241 may be the vehicle speed, steeringwheel angle, turn indicator direction, and the like detected by eachsensor provided in the vehicle. The weather information acquisition unit242 acquires weather information (Step S105). The weather informationmay be acquired from a server device provided in the MeteorologicalAgency or a weather information provider. Alternatively, the weatherinformation may be information obtained from a sensor (wiper operationdetector or raindrop detector) provided in the vehicle. The controldevice 24 may determine that the weather is rainy when the wiper isoperating or when the raindrop detector detects raindrops. Theacceleration information acquisition unit 243 acquires acceleration fromthe acceleration sensor 21 at predetermined time intervals (Step S106).The control device 24 acquires vehicle information, weather information,and acceleration at predetermined intervals.

The driving state data transmission unit 245 instructs the communicationdevice 22 to transmit the vehicle information, the weather information,and the acceleration information to the looking away determinationdevice 1 at predetermined intervals. The communication device 22transmits the vehicle information, weather information, and accelerationinformation to the looking away determination device 1 (Step S107). Thecaptured image transmission unit 246 instructs the communication device22 to transmit captured images to the looking away determination device1. The communication device 22 transmits the captured images to thelooking away determination device 1 (Step S108). The control device 24determines whether the processing is ended (Step S109), and repeats theprocessing from Step S102 until the processing is ended. The vehicleinformation, the weather information, the acceleration information, andthe captured images may be provided with the ID of the drive recorder 2,the ID of the driver, and the sensing time (the time when the driverecorder 2 performed image capture). In the aforementioned process, theacceleration and weather information are transmitted. However, whenlooking away determination is performed based only on the vehicleinformation, such information need not be transmitted. In addition tovehicle information, other information such as current positioninformation, surrounding map information, route guidance informationrelating to route guidance of the navigation system, and the like may betransmitted.

FIG. 7 is a diagram showing the processing flow of the looking awaydetermination device 1.

In the looking away determination device 1, on the basis of the ID ofthe drive recorder 2 or the ID of the driver, the informationacquisition unit 12 successively records in the database 104 a set ofthe corresponding vehicle information, weather information, accelerationinformation, and captured images in association with each ID (StepS201). Then, the control unit 11 instructs the looking away startdetermination unit 13 and the looking away determination unit 14 toperform the looking away determination process.

The looking away start determination unit 13 specifies one driverecorder 2 and acquires the sensing time, vehicle information, weatherinformation, acceleration information, and captured images that arerecorded in association with that ID. The looking away determinationunit 14 determines whether the vehicle speed included in the vehicleinformation indicates a speed in the forward direction (forwardmovement) and is a predetermined speed or higher (Step S202). Thepredetermined speed may be a value such as 20 km/h, for example. Thelooking away determination unit 14 makes a decision to start the lookingaway determination when the vehicle speed indicates forward movement andthe vehicle is at or above the predetermined speed.

The looking away start determination unit 13 may decide to start thelooking away determination by using other information or by additionallyusing other information. For example, the looking away determinationunit 14 may determine whether the acceleration is zero (0) or more, andmay decide to start the looking away determination when the accelerationis zero (0) or more. The looking away determination unit 14 may decideto start the looking away determination when the weather informationindicates rain. The looking away determination unit 14 may decide tostart the looking away determination based on a captured image outsidethe vehicle. For example, when an object appears in the straight-aheaddirection of the captured image, or when a curve appears in the lane inthe captured image, a decision to start the looking away determinationmay be made. When the looking away determination unit 14 makes thedecision to start the looking away determination, the looking awaydetermination unit 14 instructs the start. The looking awaydetermination unit 14 repeats the process of determining whether tostart the looking away determination at predetermined intervals.

The looking away determination unit 14 acquires driving stateinformation on the basis of the vehicle information (Step S203). Thedriving state information is information indicating the states duringdriving, and includes at least the vehicle speed and travel state. Thetravel state indicates the travel state of the vehicle, and includesforward movement, right turn, left turn, lane change, and curve travel.For example, the looking away determination unit 14 determines that thetravel state is forward when the direction indicator (turn indicator)does not indicate a direction and the steering wheel angle is within arange of 10 degrees to the left and right with the straight-aheaddirection serving as a standard. The looking away determination unit 14determines that the travel state is a left turn when the directionindicator is indicating the left direction and the steering wheel angledeviates 10 degrees or more to the left with the straight-aheaddirection serving as a standard. The looking away determination unit 14determines that the travel state is a right turn when the directionindicator is indicating the right direction and the steering wheel angledeviates 10 degrees or more to the right with the straight-aheaddirection serving as a standard. The looking away determination unit 14determines that the travel state is a lane change to the left when thedirection indicator indicates the left direction and the steering wheelangle is within a range of 10 degrees to the left and right with thestraight-ahead direction serving as a standard. The looking awaydetermination unit 14 determines that the travel state is a lane changeto the right when the direction indicator indicates the right directionand the steering wheel angle is within the range of 10 degrees to theright and left with the straight-ahead direction serving as a standard.The looking away determination unit 14 determines that the travel stateis left curve travel when the direction indicator does not indicate adirection and the steering wheel angle deviates 10 degrees or more tothe left with the straight-ahead direction serving as a standard. Thelooking away determination unit 14 determines that the travel state isright curve travel when the direction indicator does not indicate adirection and the steering wheel angle deviates 10 degrees or more tothe right with the straight-ahead direction serving as a standard.

The looking away determination unit 14 may determine the travel state byusing other information or additionally using other information. Forexample, upon acquiring the position information at the time of drivingfrom the drive recorder 2, the looking away determination unit 14 maydetermine the travel state on the basis of map information of theposition at the time of driving. The looking away determination unit 14may determine the travel state based on the route guidance informationwhen route guidance information of the navigation system is acquiredfrom the drive recorder 2.

The looking away determination unit 14 receives, from the informationacquisition unit 12, a captured image that has already been acquired bythe information acquisition unit 12 on the basis of the ID of the driverecorder 2 that is the current processing target. Upon having made thedecision to start the looking away determination, the looking awaydetermination unit 14 successively reads the captured images receivedfrom the drive recorder 2 at a predetermined interval.

When reading a captured image, the looking away determination unit 14performs face detection processing to determine whether a face isincluded in the captured image to an extent allowing the continuation oflooking away determination. The looking away determination unit 14determines whether or not a face can be detected in the newly acquiredcaptured image (Step S204). Upon having detected a face in the newlyacquired captured image, the looking away determination unit 14 performsthe following facial direction detection processing. When the lookingaway determination unit 14 cannot detect a face in the newly acquiredcaptured image, the looking away determination unit 14 determineswhether a face does not appear a number of times equal to or greaterthan a predetermined proportion specified on the basis of the drivingstate information, for images captured at the sensing time up to apredetermined period in the past with the sensing time of the newlyacquired captured image serving as a standard (Step S205). The lookingaway determination unit 14 specifies the predetermined proportion basedon the driving state information. For example, when the travel state isright/left turn, lane change, or curve travel, the looking awaydetermination unit 14 lowers the value of the predetermined proportionthan when the travel state is forward travel. The looking awaydetermination unit 14 determines the driver to be looking away when aface does not appear a number of times equal to or greater than thepredetermined proportion (for example, 50 percent) in captured imagesacquired during the predetermined period (for example, 1 second) (StepS206). If the number of times in which a face does not appear is lessthan the predetermined proportion for the captured images acquiredduring the predetermined period, the looking away determination unit 14does not determine that the driver is looking away, and returns to theprocess of determining whether to start the looking away determination.

In the above-described face detection process, when the driver iswearing glasses or a mask, the looking away determination unit 14 maydetermine that a face does not appear in the captured images to anextent allowing continuation of the looking away determination. In thiscase, the looking away determination unit 14 may determine to end theprocess.

In the facial direction detection processing, the looking awaydetermination unit 14 determines a condition range of the facialdirection (facial direction condition range) for looking awaydetermination based on the driving state information (Step S207). FIGS.8A to 8C are diagrams showing examples of condition ranges in the facialdirection or the line-of-sight direction for looking away determinationin the looking away determination device. For example, when the travelstate is forward, the looking away determination unit 14 decides as thecondition range a range of a degrees (for example, 10 degrees) to theleft and right with the straight-ahead direction serving as a standard,with respect to the driver D (see FIG. 8A). When the travel stateindicates a change in course to the right, the looking awaydetermination unit 14 determines the facial direction condition range tobe a range extended by β degrees in the right direction with respect tothe driver D (straight-ahead direction) more than when the travel stateis straight ahead. Examples of the case of the travel state indicating achange in course to the right direction are the travel state being aright turn, a lane change to the right lane, or right curve travel. Whenthe travel state indicates a change in course to the left direction, thelooking away determination unit 14 determines the facial directioncondition range to be a range extended by β degrees in the leftdirection with respect to the driver D (straight-ahead direction) morethan when the travel state is straight ahead. Examples of the case ofthe travel state indicating a change in course to the left direction arethe travel state being a left turn, a lane change to the left lane, orleft curve travel. The β degrees may have different values among thecase of turning right or left, the case of changing lanes, and the caseof traveling on a curve. The looking away determination unit 14 maychange the width of the condition range based on the current vehiclespeed. For example, when the speed of the vehicle is low, the driver ismore likely to see a wide range. Therefore, the looking awaydetermination unit 14 may further widen the facial direction conditionrange (width with the straight-ahead direction serving as a standard) by10 degrees to the left and right.

The looking away determination unit 14 determines whether the facialdirection in the newly acquired captured image is within the determinedpredetermined condition range (Step S208). When the facial direction inthe newly acquired captured image is within a predetermined conditionrange, the looking away determination unit 14 performs the line-of-sightdirection detection process. When the facial direction in the newlyacquired captured image is not within the predetermined condition range,the looking away determination unit 14 acquires the looking awaydetermination results of images taken up to a predetermined time in thepast (for example, 1 second) with the sensing time of the newly acquiredcaptured image serving as a standard. Based on these determinationresults, the looking away determination unit 14 determines whether thenumber of times the face is within a predetermined condition range withthe straight-ahead direction serving as a standard is equal to orgreater than a predetermined proportion (for example, 50 percent)specified based on the driving state information (Step S209). Thelooking away determination unit 14 determines there is a looking awaywhen the number of times that the face is within the predeterminedcondition range with the straight-ahead direction serving as a standardis not equal to or greater than the predetermined proportion in thecaptured images acquired during the predetermined period (Step S210).The looking away determination unit 14 changes the value of thepredetermined proportion in accordance with the driving stateinformation in the process of determining whether the number of timesthe face is within the predetermined condition range with thestraight-ahead direction serving as a standard is equal to or greaterthan the predetermined proportion. For example, when the travel state isright/left turning, lane change, or curve travel, the looking awaydetermination unit 14 lowers the value of the predetermined proportionthan when the travel state is forward travel. Moreover, the looking awaydetermination unit 14 decreases the value of the predeterminedproportion as the speed decreases. Thereby, as there is a highpossibility of visually observing a wide area when the vehicle isrunning at a low speed, it is not determined that there is a lookingaway even if visually observing a wide area. Therefore, it is possibleto reduce the unnecessary output of extra warning alerts to the driver.

In the line-of-sight direction detection processing, the looking awaydetermination unit 14 determines the condition range of theline-of-sight direction for looking away determination based on thedriving state information (Step S211). For example, when the travelstate is forward, the looking away determination unit 14 decides as thecondition range a range of a degrees (for example, 10 degrees) to theleft and right with the straight-ahead direction serving as a standard.When the travel state indicates a change in course to the rightdirection such as a right turn, a lane change to the right, or travel ona right curve, the looking away determination unit 14 determines acondition range in which the condition range during forward travel isextended by β degrees in the right direction. When the travel stateindicates a change in course to the left direction such as a left turn,a lane change to the left, or travel on a left curve, the looking awaydetermination unit 14 determines a condition range in which thecondition range during forward travel is extended by β degrees in theleft. The β degrees may have different values among the case of turningright or left, the case of changing lanes, and the case of travelling ona curve. The looking away determination unit 14 may change the width ofthe condition range on the basis of the current vehicle speed. Forexample, when the speed is low, the looking away determination unit 14may widen the width based on the straight-ahead direction by 10 degreesto the left and right with the straight-ahead direction serving as astandard since the driver is more likely to be viewing a wide range.

The looking away determination unit 14 determines whether theline-of-sight direction in the newly acquired captured image is withinthe determined predetermined condition range (line-of-sight directioncondition range) (Step S212). When the line-of-sight direction in thenewly acquired captured image is within the predetermined range, thelooking away determination unit 14 does not determine the driver to belooking away, and returns to the processing for determining whether tostart looking away determination. When the line-of-sight direction ofthe face shown in the newly acquired captured image is not within thepredetermined range, the looking away determination unit 14 acquires adetermination result for the line-of-sight of faces that appear inimages captured up to a predetermined period in the past (for example, 1second) with the sensing time of the newly acquired captured imageserving as a standard. On the basis of those determination results, thelooking away determination unit 14 determines whether or not the numberof times in which the line-of-sight is within the predeterminedcondition range with the straight-ahead direction serving as a standardis equal to or greater than a predetermined proportion specified on thebasis of the driving state information (Step S213). The looking awaydetermination unit 14 determines the driver to be looking away when thenumber of times in which the line-of-sight within the predeterminedcondition range with the straight-ahead direction serving as standard isnot equal to or greater than the predetermined proportion (for example,50 percent) for captured images acquired during the predetermined period(for example, 1 second) (Step S214). The looking away determination unit14 changes the value of the predetermined proportion in accordance withthe driving state information in the process of determining whether thenumber of times that the line-of-sight is within the predeterminedcondition range with the straight-ahead direction serving as a standardis equal to or greater than the predetermined proportion. For example,when the travel state is right/left turn, lane change or curvetraveling, the looking away determination unit 14 makes the value of thepredetermined proportion lower than when the travel state is forwardtravel. The looking away determination unit 14 decreases the value ofthe predetermined ratio as the vehicle speed decreases. Here, asdescribed above, when the speed of the vehicle is low, the driver islikely to be viewing a wide area. In such a case, even if the driverviews a wide area, the driver will not be judged as looking away. As aresult, it is possible to reduce the output of excessive looking-awayalerts to the driver.

Upon having determined that looking away is being performed, the lookingaway determination unit 14 instructs the determination result outputunit 15 to output a looking away determination result. The determinationresult output unit 15 acquires the ID of the drive recorder 2 associatedwith the data used when a determination is made that looking away isbeing performed. The determination result output unit 15 acquires thenetwork address that is the transmission destination from the database104 on the basis of the ID of the drive recorder 2. The network addressof the transmission destination is recorded in the database 104 inadvance. The determination result output unit 15 transmits informationindicating looking away detection to the transmission destination (StepS215). The determination result output unit 15 may record informationindicating the looking away detection in the database 104 in associationwith the ID of the drive recorder 2 or the ID of the driver. The lookingaway determination unit 14 determines whether to end the process (StepS216). If the process is not ended, the looking away determinationdevice 1 then repeats the same process at predetermined intervals usingthe processing received from the drive recorder 2.

The drive recorder 2 receives information indicating that looking awayof the driver has been detected. Upon receiving the information oflooking away detection, the drive recorder 2 performs a process ofnotifying the driver of the looking away detection, such as issuing analarm sound. This enables the driver to become aware of the looking awaydriving.

In the above process, the looking away determination device 1 connectedto a communication network as a cloud server performs looking awaydetermination. However, the drive recorder 2 may independently performthe above-described process of looking away determination. That is, thedrive recorder 2 may operate as the looking away determination device 1.In this case, the drive recorder 2 may exhibit the same functions as theabove-mentioned information acquisition unit 12, the looking away startdetermination unit 13, the looking away determination unit 14, and thedetermination result output unit 15. Alternatively, a vehicle-mounteddevice mounted in the vehicle and connected to the drive recorder 2 mayexhibit the same functions as the information acquisition unit 12, thelooking away start determination unit 13, the looking away determinationunit 14, and the determination result output unit 15. In this case, thevehicle-mounted device operates as the looking away determination device1.

According to the above-mentioned process, even if the facial directionor the line-of-sight direction cannot be detected, it is possible todetect looking away of the driver depending on whether or not the facecan be detected.

In addition, the above-described process outputs an alert when thefacial direction or line-of-sight direction has shifted by apredetermined proportion in a predetermined period, without performingan alert indicating that looking away has been detected just by a slightdeviation in the facial direction or line-of-sight direction. Thereby,it is possible to reduce the output of unnecessary looking awaydetection alerts to the driver.

In addition, the above-described looking away determination device 1avoids detecting the driver as looking away in a state where theline-of-sight direction deviates from the forward direction, in the caseof driving at a speed equal to or lower than a predetermined speed, suchas driving at a low speed. Thereby, it is possible to reduce the outputof unnecessary looking away detection alerts to the driver.

In addition, the above-described looking away determination device 1performs a process of broadening the range with the straight-aheaddirection serving as a standard for the case of determining apossibility of a looking away determination in response to a decrease inthe speed of the vehicle. For that reason, it is possible to performlooking away determination based on an appropriate range of the facialdirection or the line-of-sight direction according to the speed.

In addition, according to the above-described processing, in the processof determining whether the number of times in which the line-of-sight iswithin the predetermined range with the straight-ahead direction servingas a standard is equal to or greater than a predetermined proportion,the looking away determination device 1 lowers the value of thatpredetermined proportion as the speed decreases. Thereby, it is possibleto perform looking away determination based on an appropriate range ofthe facial direction or the line-of-sight direction according to thespeed.

Further, according to the above-described processing, in the process ofdetermining whether the number of times a face has been detected isequal to or greater than a predetermined proportion, the looking awaydetermination device 1 specifies a value of the predetermined proportionon the basis of the driving state information. This enablesdetermination of whether the driver is looking away not only whentraveling forward, but also when turning right or left, when changinglanes, or when traveling in a curving lane. Therefore, it is possible tomake an appropriate looking away determination according to the drivingstate.

Further, according to the above-described processing, in the process ofdetermining whether the facial direction or the line-of-sight directionis within a predetermined condition range with the straight-aheaddirection as a reference, the looking away determination device 1determines that predetermined condition range based on the driving stateinformation. Thereby, it is possible to perform the looking awaydetermination based on an appropriate range of the facial direction andline-of-sight direction according to the driving state.

Further, according to the above-described processing, in the process ofdetermining whether the number of times that the facial direction or theline-of-sight direction is within the predetermined condition range withthe straight-ahead direction serving as a standard is equal to orgreater than a predetermined proportion, the looking away determinationdevice 1 determines the predetermined proportion on the basis of thedriving state information. Thereby, it is possible to perform thelooking away determination based on an appropriate range of the facialdirection and the line-of-sight direction according to the drivingstate.

In the above example embodiment, the looking away determination device 1extends the condition range of the facial direction and theline-of-sight direction in the right direction when changing the courseto the right direction, and in the left direction when changing thecourse to the left direction, but is not limited to such aconfiguration. The entire condition range (center of the conditionrange) may be shifted to the left or right direction. For example, thelooking away determination device 1 may shift the condition range in theright direction when changing the course to the right direction, and mayshift the condition range in the left direction when changing the courseto the left direction.

FIG. 9 is a diagram showing a configuration of a looking awaydetermination device according to another example embodiment of thepresent invention.

The looking away determination device 1 only need to include at leastthe looking away determination unit 14. The looking away determinationunit 14 determines whether or not a face could be detected on the basisof the captured images. The looking away determination unit 14 makes adetermination of a looking away state when a face cannot be detected apredetermined proportion or more per unit time specified based on thedriving state information acquired during driving.

The looking away determination device 1 and the drive recorder 2described above have a computer system inside. The process of eachprocess described above is stored in a computer-readable recordingmedium in the form of a program, with the above process being performedby the computer reading and executing the program. Here, thecomputer-readable recording medium refers to a magnetic disk, amagneto-optical disk, a CD-ROM, a DVD-ROM, a semiconductor memory, orthe like. Further, the computer program may be distributed to a computervia a communication line, and the computer that receives thedistribution may execute the program.

Also, the above program may be for realizing some of the functionsdescribed above. Further, the program may be a so-called differentialfile (differential program) that can realize the functions describedabove in combination with a program already recorded in the computersystem.

While a number of example embodiments of the present invention have beendescribed above, it should be understood that they have been presentedby way of example, and do not limit the scope of the invention. It willbe understood that various additions, omissions, substitutions andmodifications may be made to these example embodiments within the scopenot departing from the spirit of the invention.

The whole or part of the exemplary example embodiments disclosed abovecan be described as, but not limited to, the following supplementarynotes.

(Supplementary Note 1)

A looking away determination device comprising:

a determination unit that determines that a driver is in a looking awaystate when a proportion of an image in which a face of the driver is notdetected with respect to a plurality of images obtained by imaging thedriver is equal to or greater than a first predetermined value, thefirst predetermined value being determined based on driving stateinformation that indicates a state during driving by the driver.

(Supplementary Note 2)

The looking away determination device according to supplementary note 1,wherein the determination unit determines that the driver is in thelooking away state when a proportion of an image in which a facialdirection of the driver is not within a facial direction condition rangewith respect to the plurality of images obtained by imaging the driveris equal to or greater than a second predetermined value, the facialdirection condition range being determined based on the driving stateinformation.

(Supplementary Note 3)

The looking away determination device according to supplementary note 2,wherein the determination unit determines that the driver is in thelooking away state when a proportion of an image in which aline-of-sight direction of the driver is not within a line-of-sightdirection condition range with respect to the plurality of imagesobtained by imaging the driver is equal to or greater than a thirdpredetermined value, the line-of-sight direction condition range beingdetermined based on the driving state information.

(Supplementary Note 4)

The looking away determination device according to supplementary note 2or supplementary note 3, wherein the determination unit extends thefacial direction condition range in a right direction when the drivingstate information indicates that a course of a vehicle driven by thedriver is changed to the right direction.

(Supplementary Note 5)

The looking away determination device according to supplementary note 3or supplementary note 4, wherein the determination unit extends theline-of-sight direction condition range to a right direction when thedriving state information indicates that a course of a vehicle driven bythe driver is changed in the right direction.

(Supplementary Note 6)

The looking away determination device according to supplementary note 2or supplementary note 3, wherein the determination unit extends thefacial direction condition range in a left direction when the drivingstate information indicates that a course of a vehicle driven by thedriver is changed to the left direction.

(Supplementary Note 7)

The looking away determination device according to supplementary note 3or supplementary note 6, wherein the determination unit extends theline-of-sight direction condition range in a left direction when thedriving state information indicates that a course of a vehicle driven bythe driver is changed to the left direction.

(Supplementary Note 8)

The looking away determination device according to supplementary note 1,wherein the determination unit determines the first predetermined valuebased on the driving state information.

(Supplementary Note 9)

The looking away determination device according to supplementary note 2,wherein the determination unit determines the facial direction conditionrange based on the driving state information.

(Supplementary Note 10)

The looking away determination device according to supplementary note 3,wherein the determination unit determines the line-of-sight directioncondition range based on the driving state information.

(Supplementary Note 11)

The looking away determination device according to supplementary note 1,wherein the determination unit detects the face of the driver from eachof the plurality of images.

(Supplementary Note 12)

The looking away determination device according to supplementary note 1,

wherein the determination unit detects the face of the driver from afirst image captured last among the plurality of images, and

the determination unit detects the face of the driver from each of theplurality of images when the face of the driver is not detected from thefirst image.

(Supplementary Note 13)

The looking away determination device according to supplementary note12,

wherein the determination unit detects a facial direction of the driverfrom the first image when the face of the driver is detected from thefirst image, and

the determination unit determines whether the detected facial directionis within a facial direction condition range.

(Supplementary Note 14)

The looking away determination device according to supplementary note13,

wherein when the detected facial direction is not determined to bewithin the facial direction condition range, the determination unitdetects the facial direction of the driver from each of the plurality ofimages, and

the determination unit determines that the driver is in the looking awaystate when the proportion of an image in which the facial direction ofthe driver is not within the facial direction condition range withrespect to the plurality of images is equal to or greater than a secondpredetermined value.

(Supplementary Note 15)

The looking away determination device according to supplementary note 13or supplementary note 14,

wherein the determination unit detects a line-of-sight direction of thedriver from the first image when the detected facial direction isdetermined to be within the facial direction condition range, and

the determination unit determines whether the detected line-of-sightdirection is within a line-of-sight direction condition range.

(Supplementary Note 16)

The looking away determination device according to supplementary note15,

wherein when the detected line-of-sight direction is not determined tobe within the line-of-sight direction condition range, the determinationunit detects the line-of-sight direction of the driver from each of theplurality of images, and

the determination unit determines that the driver is in the looking awaystate when a proportion of an image in which the line-of-sight directionof the driver is not within the line-of-sight direction condition rangewith respect to the plurality of images is equal to or greater than athird predetermined value.

(Supplementary Note 17)

A looking away determination system comprising an imaging device and alooking away determination device,

wherein the looking away determination device comprises a determinationunit that determines that a driver is in a looking away state when aproportion of an image in which a face of the driver is not detectedwith respect to a plurality of images obtained by imaging the driver bythe imaging device is equal to or greater than a first predeterminedvalue, the first predetermined value being determined based on drivingstate information that indicates a state during driving by the driver.

(Supplementary Note 18)

A looking away determination method comprising:

determining that a driver is in a looking away state when a proportionof an image in which a face of the driver is not detected with respectto a plurality of images obtained by imaging the driver is equal to orgreater than a first predetermined value, the first predetermined valuebeing determined based on driving state information that indicates astate during driving by the driver.

(Supplementary Note 19)

A storage medium that stores a program that causes a computer toexecute:

determining that a driver is in a looking away state when a proportionof an image in which a face of the driver is not detected with respectto a plurality of images obtained by imaging the driver is equal to orgreater than a first predetermined value, the first predetermined valuebeing determined based on driving state information that indicates astate during driving by the driver.

This application is based upon and claims the benefit of priority fromJapanese Patent Application No. 2018-59161, filed Mar. 27, 2018, thedisclosure of which is incorporated herein in its entirety.

INDUSTRIAL APPLICABILITY

The present invention may be applied to a looking away determinationdevice, a looking away determination system, a looking awaydetermination method, and a storage medium.

REFERENCE SYMBOLS

-   -   1: Looking away determination device    -   2: Drive recorder    -   11: Control unit    -   12: Information acquisition unit    -   13: Looking away start determination unit    -   14: Looking away determination unit    -   15: Determination result output unit    -   21: Acceleration sensor    -   23: Camera    -   24: Control device    -   241: Vehicle information acquisition unit    -   242: Weather information acquisition unit    -   223: Acceleration information acquisition unit    -   244: Captured image acquisition unit    -   245: Driving state data transmission unit    -   246: Captured image transmission unit

1. A looking away determination device comprising: a memory configuredto store instructions; and a processor configured to execute theinstructions to: determine that a driver is in a looking away state whena proportion of an image in which a face of the driver is not detectedwith respect to a plurality of images obtained by imaging the driver isequal to or greater than a first predetermined value, the firstpredetermined value being determined based on driving state informationthat indicates a state during driving by the driver.
 2. The looking awaydetermination device according to claim 1, wherein the processor isconfigured to execute the instructions to: determine that the driver isin the looking away state when a proportion of an image in which afacial direction of the driver is not within a facial directioncondition range with respect to the plurality of images obtained byimaging the driver is equal to or greater than a second predeterminedvalue, the facial direction condition range being determined based onthe driving state information.
 3. The looking away determination deviceaccording to claim 2, wherein the processor is configured to execute theinstructions to: determine that the driver is in the looking away statewhen a proportion of an image in which a line-of-sight direction of thedriver is not within a line-of-sight direction condition range withrespect to the plurality of images obtained by imaging the driver isequal to or greater than a third predetermined value, the line-of-sightdirection condition range being determined based on the driving stateinformation.
 4. The looking away determination device according to claim2, wherein the processor is configured to execute the instructions to:extend the facial direction condition range in a right direction whenthe driving state information indicates that a course of a vehicledriven by the driver is changed to the right direction.
 5. The lookingaway determination device according to claim 3, wherein the processor isconfigured to execute the instructions to: extend the line-of-sightdirection condition range to a right direction when the driving stateinformation indicates that a course of a vehicle driven by the driver ischanged in the right direction.
 6. The looking away determination deviceaccording to claim 2, wherein the processor is configured to execute theinstructions to: extend the facial direction condition range in a leftdirection when the driving state information indicates that a course ofa vehicle driven by the driver is changed to the left direction.
 7. Thelooking away determination device according to claim 3, wherein theprocessor is configured to execute the instructions to: extend theline-of-sight direction condition range in a left direction when thedriving state information indicates that a course of a vehicle driven bythe driver is changed to the left direction.
 8. The looking awaydetermination device according to claim 1, wherein the processor isconfigured to execute the instructions to: determine the firstpredetermined value based on the driving state information.
 9. Thelooking away determination device according to claim 2, wherein theprocessor is configured to execute the instructions to: determine thefacial direction condition range based on the driving state information.10. The looking away determination device according to claim 3, whereinthe processor is configured to execute the instructions to: determinethe line-of-sight direction condition range based on the driving stateinformation.
 11. The looking away determination device according toclaim 1, wherein the processor is configured to execute the instructionsto: try to detect the face of the driver from each of the plurality ofimages.
 12. The looking away determination device according to claim 1,try to detect the face of the driver from a first image captured lastamong the plurality of images; and try to detect the face of the driverfrom each of the plurality of images when the face of the driver is notdetected from the first image.
 13. The looking away determination deviceaccording to claim 12, wherein the processor is configured to executethe instructions to: detect a facial direction of the driver from thefirst image when the face of the driver is detected from the firstimage; and determine whether the detected facial direction is within afacial direction condition range.
 14. The looking away determinationdevice according to claim 13, wherein the processor is configured toexecute the instructions to: detect, when the detected facial directionis not determined to be within the facial direction condition range, thefacial direction of the driver from each of the plurality of images; anddetermine that the driver is in the looking away state when theproportion of an image in which the facial direction of the driver isnot within the facial direction condition range with respect to theplurality of images is equal to or greater than a second predeterminedvalue.
 15. The looking away determination device according to claim 13,wherein the processor is configured to execute the instructions to:detect a line-of-sight direction of the driver from the first image whenthe detected facial direction is determined to be within the facialdirection condition range; and determine whether the detectedline-of-sight direction is within a line-of-sight direction conditionrange.
 16. The looking away determination device according to claim 15,wherein the processor is configured to execute the instructions to:detect, when the detected line-of-sight direction is not determined tobe within the line-of-sight direction condition range the line-of-sightdirection of the driver from each of the plurality of images; anddetermine that the driver is in the looking away state when a proportionof an image in which the line-of-sight direction of the driver is notwithin the line-of-sight direction condition range with respect to theplurality of images is equal to or greater than a third predeterminedvalue.
 17. A looking away determination system comprising: the lookingaway determination device according to claim 1; and a camera thatobtains the plurality of images by imaging.
 18. A looking awaydetermination method comprising: determining that a driver is in alooking away state when a proportion of an image in which a face of thedriver is not detected with respect to a plurality of images obtained byimaging the driver is equal to or greater than a first predeterminedvalue, the first predetermined value being determined based on drivingstate information that indicates a state during driving by the driver.19. A non-transitory computer-readable storage medium that stores aprogram that causes a computer to execute: determining that a driver isin a looking away state when a proportion of an image in which a face ofthe driver is not detected with respect to a plurality of imagesobtained by imaging the driver is equal to or greater than a firstpredetermined value, the first predetermined value being determinedbased on driving state information that indicates a state during drivingby the driver.